Eye tracking system for head-mounted display devices

ABSTRACT

The present disclosure is directed to systems and methods of eye tracking for use in various applications, such as virtual reality or augmented reality applications that include head-mounted display devices. An eye tracking subsystem may be provided that includes a plurality of assemblies that each include a light source, such as an LED, a light detector, such as a silicon photodiode, and a polarizer that is configured to prevent light reflected via specular reflection from reaching the light detectors so that only scattered light is detected by the light detectors. Machine learning or other techniques may be used to track or otherwise determine a user&#39;s gaze direction, which may be used by one or more components of an HMD device to improve its functionality in various ways.

TECHNICAL FIELD

The following disclosure relates generally to techniques for eyetracking, and more specifically to techniques for eye tracking for usein head-mounted display devices.

BACKGROUND

A head-mounted display (HMD) device or system is an electronic devicethat is worn on a user's head and, when so worn, secures at least oneelectronic display within a viewable field of at least one of the user'seyes, regardless of the position or orientation of the user's head. HMDdevices used to implement virtual reality systems typically envelop awearer's eyes completely and substitute a “virtual” reality for theactual view (or actual reality) in front of the wearer, while HMDdevices for augmented reality systems typically provide asemi-transparent or transparent overlay of one or more screens in frontof a wearer's eyes such that an actual view is augmented with additionalinformation. For augmented reality systems, the “display” component of aHMD device is either transparent or at a periphery of the user's fieldof view so that it does not completely block the user from being able tosee their external environment.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of a networked environment that includesone or more systems suitable for performing at least some techniquesdescribed in the present disclosure, including embodiments of an eyetracking subsystem.

FIG. 2 is a diagram illustrating an example environment in which atleast some of the described techniques are used with an examplehead-mounted display device that is tethered to a video renderingcomputing system and providing a virtual reality display to a user.

FIG. 3 is a front pictorial diagram of an HMD device having binoculardisplay subsystems.

FIG. 4 illustrates a top plan view of an HMD device having binoculardisplay subsystems and various sensors, according to an exampleembodiment of the present disclosure.

FIG. 5 illustrates an example of using light sources and light detectorsto determine pupil location, such as for use in eye tracking in HMDdevices in accordance with the described techniques of the presentdisclosure.

FIG. 6 is a schematic diagram of an environment in which machinelearning techniques may be used to implement an eye tracking subsystemof an HMD device, according to one non-limiting illustratedimplementation.

FIG. 7 is a flow diagram for a method of operating an HMD device thatincludes eye tracking capabilities, according to one non-limitingillustrated implementation.

BRIEF SUMMARY

A head-mounted display (HMD) system may be summarized as including asupport structure wearable on the head of a user; an eye trackingsubsystem coupled to the support structure, the eye tracking subsystemincluding a plurality of eye tracking assemblies that each include alight source operative to emit light; a light detector operative todetect light; and a polarizer positioned proximate to at least one ofthe light source and the light detector, the polarizer configured toprevent light reflected via specular reflection from being received bythe light detector; at least one processor; and memory storing a set ofinstructions or data that, as a result of execution, cause the HMDsystem to selectively cause the light sources of the plurality of eyetracking assemblies to emit light; receive light detection informationcaptured by the light detectors of the plurality of eye trackingassemblies; provide the received light detection information as input toa prediction model; receive, from the prediction model in response toproviding the light detection information, a determined gaze directionfor an eye of the user; and provide the determined gaze direction to acomponent associated with the HMD system for use thereby. The lightdetection information may include a signature radiation pattern of eachof the light sources after the light from the light source hasreflected, scattered, or been absorbed of off the face or eye of theuser. Each of the light sources may be directed toward an expectedlocation of a pupil of the user. The light sources may include lightemitting diodes that emit light having a wavelength that is between 780nm and 1000 nm. The light detectors may include photodiodes. The eyetracking subsystem may include four eye tracking assemblies positionedto determine the gaze direction of a left eye of the user and four eyetracking assemblies positioned to determine the gaze direction of aright eye of the user. The polarizer may include two cross linearpolarizers. For each of the plurality of eye tracking assemblies, thepolarizer may include a first polarizer positioned in a light emittingpath of the light source and a second polarizer positioned in a lightdetecting path of the light detector. The polarizer may include at leastone of a circular polarizer or a linear polarizer. For each of theplurality of eye tracking assemblies, the light source may be positionedaway from an optical axis of an eye of the user to provide dark fieldillumination of the pupil. The prediction model may include a machinelearning model or other type of function or model (e.g., polynomial(s),lookup table). The machine learning model may include a mixture densitynetwork (MDN) model. The machine learning model may include a recurrentneural network (RNN) model. The machine learning model may utilize pastinput information or eye motion information to determine the gazedirection. The machine learning model may be a model trained duringfield operation of a plurality of HMD systems. The HMD system mayinclude at least one display, and the at least one processor may causethe at least one display to present a user interface element; mayselectively cause the light sources of the plurality of eye trackingassemblies to emit light; may receive light detection informationcaptured by the light detectors of the plurality of eye trackingassemblies; and may update the machine learning model based at least inpart on the received light detection information and known or inferredgaze direction information associated with the received light detectioninformation. The user interface element may include a static userinterface element or a moving user interface element. The HMD system mayinclude at least one display, and the at least one processor maydynamically modify a rendering output of the at least one display basedat least in part on the determined gaze direction. The HMD system mayinclude an interpupillary distance (IPD) adjustment component, and theat least one processor may cause the IPD adjustment component to alignat least one component of the HMD system for the user based at least inpart on the determined gaze direction.

A method of operating a head-mounted display (HMD) system, the HMDsystem may include an eye tracking subsystem coupled to a supportstructure that includes a plurality of eye tracking assemblies that eachinclude a light source, a light detector, and a polarizer, the methodmay be summarized as including selectively causing the light sources ofthe plurality of eye tracking assemblies to emit light; receiving lightdetection information captured by the plurality of light detectors;providing the received light detection information as input to a trainedmachine learning model; receiving, from the machine learning model inresponse to providing the light detection information, a determined gazedirection for an eye of a user; and providing the determined gazedirection to a component associated with the HMD system for use thereby.Providing the received light detection information as input to a trainedmachine learning model may include providing the received lightdetection information as input to a mixture density network (MDN) model.Providing the received light detection information as input to a trainedmachine learning model may include providing the received lightdetection information as input to a recurrent neural network (RNN)model. Providing the received light detection information as input to atrained machine learning model may include providing the received lightdetection information to a machine learning model that utilizes pastinput information or eye motion information to determine the gazedirection.

The method may further include training the machine learning modelduring field operation of a plurality of HMD systems. The HMD system mayinclude at least one display, and the method may include causing the atleast one display to present a user interface element; selectivelycausing the light sources of the plurality of eye tracking assemblies toemit light; receiving light detection information captured by theplurality of light detectors; and updating the machine learning modelbased at least in part on the received light detection information andknown or inferred gaze direction information associated with thereceived light detection information. Causing the at least one displayto present a user interface element may include causing the at least onedisplay to present a static user interface element or a moving userinterface element. The HMD system may include at least one display, andthe method may include dynamically modifying an output of the at leastone display based at least in part on the determined gaze direction.

The method may further include mechanically aligning at least onecomponent of the HMD system for the user based at least in part on thedetermined gaze direction.

A head-mounted display (HMD) system may be summarized as including asupport structure wearable on the head of a user; an eye trackingsubsystem coupled to the support structure, the eye tracking subsystemincluding a plurality of eye tracking assemblies that each include alight emitting diode; a photodiode; and a polarizer positioned proximateto at least one of the light emitting diode and the photodiode, thepolarizer configured to prevent light reflected via specular reflectionfrom being received by the photodiode; at least one processor; andmemory storing a set of instructions or data that, as a result ofexecution, cause the HMD system to selectively cause the light emittingdiodes of the plurality of eye tracking assemblies to emit light;receive light detection information captured by the photodiodes of theplurality of eye tracking assemblies;

provide the received light detection information as input to a trainedmachine learning model; receive, from the machine learning model inresponse to providing the light detection information, a determined gazedirection for an eye of the user; and dynamically modify operation of acomponent associated with the HMD system based at least in part on thedetermined gaze direction.

DETAILED DESCRIPTION

In the following description, certain specific details are set forth inorder to provide a thorough understanding of various disclosedimplementations. However, one skilled in the relevant art will recognizethat implementations may be practiced without one or more of thesespecific details, or with other methods, components, materials, etc. Inother instances, well-known structures associated with computer systems,server computers, and/or communications networks have not been shown ordescribed in detail to avoid unnecessarily obscuring descriptions of theimplementations.

Unless the context requires otherwise, throughout the specification andclaims that follow, the word “comprising” is synonymous with“including,” and is inclusive or open-ended (i.e., does not excludeadditional, unrecited elements or method acts).

Reference throughout this specification to “one implementation” or “animplementation” means that a particular feature, structure orcharacteristic described in connection with the implementation isincluded in at least one implementation. Thus, the appearances of thephrases “in one implementation” or “in an implementation” in variousplaces throughout this specification are not necessarily all referringto the same implementation. Furthermore, the particular features,structures, or characteristics may be combined in any suitable manner inone or more implementations.

As used in this specification and the appended claims, the singularforms “a,” “an,” and “the” include plural referents unless the contextclearly dictates otherwise. It should also be noted that the term “or”is generally employed in its sense including “and/or” unless the contextclearly dictates otherwise.

The headings and Abstract of the Disclosure provided herein are forconvenience only and do not interpret the scope or meaning of theimplementations.

Eye tracking is a process by which the position, orientation, or motionof the eye may be measured, detected, sensed, determined, or monitored(collectively, “measured”). In many applications, this is done with aview towards determining the gaze direction of a user. The position,orientation, or motion of the eye may be measured in a variety ofdifferent ways, the least invasive of which may employ one or moreoptical detectors or sensors to optically track the eye. Some techniquesmay involve illuminating or flooding the entire eye, all at once, withinfrared light and measuring reflections with at least one opticalsensor that is tuned to be sensitive to the infrared light. Informationabout how the infrared light is reflected from the eye is analyzed todetermine the positions, orientations, and/or motions of one or more eyefeatures such as the cornea, pupil, iris, or retinal blood vessels.

Eye tracking functionality is highly advantageous in applications ofwearable head-mounted display systems. Some examples of the utility ofeye tracking in head-mounted display system include influencing wherecontent is displayed in the user's field of view, conserving power,bandwidth, or computational resources by modifying the display ofcontent that is outside of the user's field of view (e.g., foveatedrendering), influencing what content is displayed to the user,determining where the user is looking or gazing, determining whether theuser is looking at displayed content on a display, providing a methodthrough which the user may control or interact with displayed content,and other applications.

The present disclosure relates generally to techniques for eye tracking.Such techniques may be used, for example, in a head-mounted display(“HMD”) device used for VR or AR applications. Some or all of thetechniques described herein may be performed via automated operations ofembodiments of an eye tracking subsystem, such as implemented by one ormore configured hardware processors and/or other configured hardwarecircuitry. The one or more hardware processors or other configuredhardware circuitry of such a system or device may include, for example,one or more GPUs (“graphical processing units”) and/or CPUs (“centralprocessing units”) and/or other microcontrollers (“MCUs”) and/or otherintegrated circuits, such as with the hardware processor(s) being partof an HMD device or other device that incorporates one or more displaypanels on which the image data will be displayed or being part of acomputing system that generates or otherwise prepares the image data tobe sent to the display panel(s) for display, as discussed further below.More generally, such a hardware processors or other configured hardwarecircuitry may include, but are not limited to, one or moreapplication-specific integrated circuits (ASICs), standard integratedcircuits, controllers (e.g., by executing appropriate instructions, andincluding microcontrollers and/or embedded controllers),field-programmable gate arrays (FPGAs), complex programmable logicdevices (CPLDs), digital signal processors (DSPs), programmable logiccontrollers (PLCs), etc. Additional details are included elsewhereherein, including with respect to FIG. 1 discussed below.

Technical benefits in at least some embodiments of the describedtechniques include addressing and mitigating increased mediatransmission bandwidths for image encoding by reducing image data size,improving speed of controlling display panel pixels (e.g., based atleast in part on corresponding reduced image data size), improvingfoveated image systems and other techniques that reflect subsets ofdisplay panels and/or images of particular interest, etc. Foveated imageencoding systems take advantage of particular aspects of the humanvisual system (which may provide detailed information only at and arounda point of focus), but often use specialized computational processing inorder to avoid visual artifacts to which peripheral vision is verysensitive (e.g., artifacts related to motion and contrast in video andimage data). In cases of certain VR and AR displays, both the bandwidthand computing usage for processing high resolution media are exacerbatedbecause a particular display device involves two separate display panels(i.e., one for each eye) with two separately addressable pixel arrays,each involving an appropriate resolution. Thus, the described techniquesmay be used, for example, for decreasing the transmission bandwidth forlocal and/or remote display of a video frame or other image, whilepreserving resolution and detail in a viewer's “area of interest” withinan image while also minimizing computing usage for processing such imagedata. Furthermore, the use of lenses in head-mounted display devices andwith other displays may provide a greater focus or resolution on asubset of the display panel, such that using such techniques to displaylower-resolution information in other portions of the display panel mayfurther provide benefits when using such techniques in such embodiments.

For illustrative purposes, some embodiments are described below in whichspecific types of information are acquired and used in specific types ofways for specific types of structures and by using specific types ofdevices. However, it will be understood that such described techniquesmay be used in other manners in other embodiments, and that the presentdisclosure is thus not limited to the exemplary details provided. As onenon-exclusive example, various of the embodiments discussed hereininclude the use of images that are video frames—however, while manyexamples described herein refer to a “video frame” for convenience, itwill be appreciated that the techniques described with reference to suchexamples may be employed with respect to one or more images of varioustypes, including non-exclusive examples of multiple video frames insuccession (e.g., at 30, 60, 90, 180 or some other quantity of framesper second), other video content, photographs, computer-generatedgraphical content, other articles of visual media, or some combinationthereof. In addition, various details are provided in the drawings andtext for exemplary purposes, but are not intended to limit the scope ofthe present disclosure. In addition, as used herein, a “pixel” refers tothe smallest addressable image element of a display that may beactivated to provide all possible color values for that display. In manycases, a pixel includes individual respective sub-elements (in somecases as separate “sub-pixels”) for separately producing red, green, andblue light for perception by a human viewer, with separate colorchannels used to encode pixel values for the sub-pixels of differentcolors. A pixel “value” as used herein refers to a data valuecorresponding to respective levels of stimulation for one or more ofthose respective RGB elements of a single pixel.

FIG. 1 is a schematic diagram of a networked environment 100 thatincludes a local media rendering (LMR) system 110 (e.g., a gamingsystem), which includes a local computing system 120 and display device180 (e.g., an HMD device with two display panels) suitable forperforming at least some techniques described herein. In the depictedembodiment of FIG. 1, the local computing system 120 is communicativelyconnected to display device 180 via transmission link 115 (which may bewired or tethered, such as via one or more cables as illustrated in FIG.2 (cable 220), or instead may be wireless). In other embodiments, thelocal computing system 120 may provide encoded image data for display toa panel display device (e.g., a TV, console or monitor) via a wired orwireless link, whether in addition to or instead of the HMD device 180,and the display devices each includes one or more addressable pixelarrays. In various embodiments, the local computing system 120 mayinclude a general purpose computing system; a gaming console; a videostream processing device; a mobile computing device (e.g., a cellulartelephone, PDA, or other mobile device); a VR or AR processing device;or other computing system.

In the illustrated embodiment, the local computing system 120 hascomponents that include one or more hardware processors (e.g.,centralized processing units, or “CPUs”) 125, memory 130, various I/O(“input/output”) hardware components 127 (e.g., a keyboard, a mouse, oneor more gaming controllers, speakers, microphone, IR transmitter and/orreceiver, etc.), a video subsystem 140 that includes one or morespecialized hardware processors (e.g., graphics processing units, or“GPUs”) 144 and video memory (VRAM) 148, computer-readable storage 150,and a network connection 160. Also in the illustrated embodiment, anembodiment of an eye tracking subsystem 135 executes in memory 130 inorder to perform at least some of the described techniques, such as byusing the CPU(s) 125 and/or GPU(s) 144 to perform automated operationsthat implement those described techniques, and the memory 130 mayoptionally further execute one or more other programs 133 (e.g., togenerate video or other images to be displayed, such as a game program).As part of the automated operations to implement at least sometechniques described herein, the eye tracking subsystem 135 and/orprograms 133 executing in memory 130 may store or retrieve various typesof data, including in the example database data structures of storage150, in this example, the data used may include various types of imagedata information in database (“DB”) 154, various types of applicationdata in DB 152, various types of configuration data in DB 157, and mayinclude additional information, such as system data or otherinformation.

The LMR system 110 is also, in the depicted embodiment, communicativelyconnected via one or more computer networks 101 and network links 102 toan exemplary network-accessible media content provider 190 that mayfurther provide content to the LMR system 110 for display, whether inaddition to or instead of the image-generating programs 133. The mediacontent provider 190 may include one or more computing systems (notshown) that may each have components similar to those of local computingsystem 120, including one or more hardware processors, I/O components,local storage devices and memory, although some details are notillustrated for the network-accessible media content provider for thesake of brevity.

It will be appreciated that, while the display device 180 is depicted asbeing distinct and separate from the local computing system 120 in theillustrated embodiment of FIG. 1, in certain embodiments some or allcomponents of the local media rendering system 110 may be integrated orhoused within a single device, such as a mobile gaming device, portableVR entertainment system, HMD device, etc. In such embodiments,transmission link 115 may, for example, include one or more system busesand/or video bus architectures.

As one example involving operations performed locally by the local mediarendering system 120, assume that the local computing system is a gamingcomputing system, such that application data 152 includes one or moregaming applications executed via CPU 125 using memory 130, and thatvarious video frame display data is generated and/or processed by theimage-generating programs 133, such as in conjunction with GPU 144 ofthe video subsystem 140. In order to provide a quality gamingexperience, a high volume of video frame data (corresponding to highimage resolution for each video frame, as well as a high “frame rate” ofapproximately 60-180 of such video frames per second) is generated bythe local computing system 120 and provided via the wired or wirelesstransmission link 115 to the display device 180.

It will also be appreciated that computing system 120 and display device180 are merely illustrative and are not intended to limit the scope ofthe present disclosure. The computing system 120 may instead includemultiple interacting computing systems or devices, and may be connectedto other devices that are not illustrated, including through one or morenetworks such as the Internet, via the Web, or via private networks(e.g., mobile communication networks, etc.). More generally, a computingsystem or other computing node may include any combination of hardwareor software that may interact and perform the described types offunctionality, including, without limitation, desktop or othercomputers, game systems, database servers, network storage devices andother network devices, PDAs, cell phones, wireless phones, pagers,electronic organizers, Internet appliances, television-based systems(e.g., using set-top boxes and/or personal/digital video recorders), andvarious other consumer products that include appropriate communicationcapabilities. The display device 180 may similarly include one or moredevices with one or more display panels of various types and forms, andoptionally include various other hardware and/or software components.

In addition, the functionality provided by the eye tracking subsystem135 may in some embodiments be distributed in one or more components,and in some embodiments some of the functionality of the eye trackingsubsystem 135 may not be provided and/or other additional functionalitymay be available. It will also be appreciated that, while various itemsare illustrated as being stored in memory or on storage while beingused, these items or portions of them may be transferred between memoryand other storage devices for purposes of memory management or dataintegrity. Thus, in some embodiments, some or all of the describedtechniques may be performed by hardware that include one or moreprocessors or other configured hardware circuitry or memory or storage,such as when configured by one or more software programs (e.g., by theeye tracking subsystem 135 or it components) and/or data structures(e.g., by execution of software instructions of the one or more softwareprograms and/or by storage of such software instructions and/or datastructures). Some or all of the components, systems and data structuresmay also be stored (e.g., as software instructions or structured data)on a non-transitory computer-readable storage medium, such as a harddisk or flash drive or other non-volatile storage device, volatile ornon-volatile memory (e.g., RAM), a network storage device, or a portablemedia article to be read by an appropriate drive (e.g., a DVD disk, a CDdisk, an optical disk, etc.) or via an appropriate connection. Thesystems, components and data structures may also in some embodiments betransmitted as generated data signals (e.g., as part of a carrier waveor other analog or digital propagated signal) on a variety ofcomputer-readable transmission mediums, including wireless-based andwired/cable-based mediums, and may take a variety of forms (e.g., aspart of a single or multiplexed analog signal, or as multiple discretedigital packets or frames). Such computer program products may also takeother forms in other embodiments. Accordingly, the present invention maybe practiced with other computer system configurations.

FIG. 2 illustrates an example environment 200 in which at least some ofthe described techniques are used with an example HMD device 202 that iscoupled to a video rendering computing system 204 via a tetheredconnection 220 (or a wireless connection in other embodiments) toprovide a virtual reality display to a human user 206. The user wearsthe HMD device 202 and receives displayed information via the HMD devicefrom the computing system 204 of a simulated environment different fromthe actual physical environment, with the computing system acting as animage rendering system that supplies images of the simulated environmentto the HMD device for display to the user, such as images generated by agame program and/or other software program executing on the computingsystem. The user is further able to move around within a tracked volume201 of the actual physical environment 200 in this example, and mayfurther have one or more I/O (“input/output”) devices to allow the userto further interact with the simulated environment, which in thisexample includes hand-held controllers 208 and 210.

In the illustrated example, the environment 200 may include one or morebase stations 214 (two shown, labeled base stations 214 a and 214 b)that may facilitate tracking of the HMD device 202 or the controllers208 and 210. As the user moves location or changes orientation of theHMD device 202, the position of the HMD device is tracked, such as toallow a corresponding portion of the simulated environment to bedisplayed to the user on the HMD device, and the controllers 208 and 210may further employ similar techniques to use in tracking the positionsof the controllers (and to optionally use that information to assist indetermining or verifying the position of the HMD device). After thetracked position of the HMD device 202 is known, correspondinginformation is transmitted to the computing system 204 via the tether220 or wirelessly, which uses the tracked position information togenerate one or more next images of the simulated environment to displayto the user.

There are numerous different methods of positional tracking that may beused in the various implementations of the present disclosure,including, but not limited to, acoustic tracking, inertial tracking,magnetic tracking, optical tracking, combinations thereof, etc.

In at least some implementations, the HMD device 202 may include one ormore optical receivers or sensors that may be used to implement trackingfunctionality or other aspects of the present disclosure. For example,the base stations 214 may each sweep an optical signal across thetracked volume 201. Depending on the requirements of each particularimplementation, each base station 214 may generate more than one opticalsignal. For example, while a single base station 214 is typicallysufficient for six-degree-of-freedom tracking, multiple base stations(e.g., base stations 214 a, 214 b) may be necessary or desired in someembodiments to provide robust room-scale tracking for HMD devices andperipherals. In this example, optical receivers are incorporated intothe HMD device 202 and or other tracked objects, such as the controllers208 and 210. In at least some implementations, optical receivers may bepaired with an accelerometer and gyroscope Inertial Measurement Unit(“IMU”) on each tracked device to support low-latency sensor fusion.

In at least some implementations, each base station 214 includes tworotors which sweep a linear beam across the tracked volume 201 onorthogonal axes. At the start of each sweep cycle, the base station 214may emit an omni-directional light pulse (referred to as a “syncsignal”) that is visible to all sensors on the tracked objects. Thus,each sensor computes a unique angular location in the swept volume bytiming the duration between the sync signal and the beam signal. Sensordistance and orientation may be solved using multiple sensors affixed toa single rigid body.

The one or more sensors positioned on the tracked objects (e.g., HMDdevice 202, controllers 208 and 210) may comprise an optoelectronicdevice capable of detecting the modulated light from the rotor. Forvisible or near-infrared (NIR) light, silicon photodiodes and suitableamplifier/detector circuitry may be used. Because the environment 200may contain static and time-varying signals (optical noise) with similarwavelengths to the signals of the base stations 214 signals, in at leastsome implementations the base station light may be modulated in such away as to make it easy to differentiate from any interfering signals,and/or to filter the sensor from any wavelength of radiation other thanthat of base station signals.

Inside-out tracking is also a type positional tracking that may be usedto track the position of the HMD device 202 and/or other objects (e.g.,controllers 208 and 210, tablet computers, smartphones). Inside-outtracking differs from outside-in tracking by the location of the camerasor other sensors used to determine the HMD's position. For inside-outtracking, the camera or sensors are located on the HMD, or object beingtracked, while in outside-out tracking the camera or sensors are placedin a stationary location in the environment.

An HMD that utilizes inside-out tracking utilizes one or more cameras to“look out” to determine how its position changes in relation to theenvironment. When the HMD moves, the sensors readjust their place in theroom and the virtual environment responds accordingly in real-time. Thistype of positional tracking can be achieved with or without markersplaced in the environment. The cameras that are placed on the HMDobserve features of the surrounding environment. When using markers, themarkers are designed to be easily detected by the tracking system andplaced in a specific area. With “markerless” inside-out tracking, theHMD system uses distinctive characteristics (e.g., natural features)that originally exist in the environment to determine position andorientation. The HMD system's algorithms identify specific images orshapes and use them to calculate the device's position in space. Datafrom accelerometers and gyroscopes can also be used to increase theprecision of positional tracking.

FIG. 3 shows information 300 illustrating a front view of an example HMDdevice 344 when worn on the head of a user 342. The HMD device 344includes a front-facing structure 343 that supports a front-facing orforward camera 346 and a plurality of sensors 348 a-348 d (collectively348) of one or more types. As one example, some or all of the sensors348 may assist in determining the location and orientation of the device344 in space, such as light sensors to detect and use light informationemitted from one or more external devices (not shown, e.g., basestations 214 of FIG. 2). As shown, the forward camera 346 and thesensors 348 are directed forward toward an actual scene or environment(not shown) in which the user 342 operates the HMD device 344. Theactual physical environment may include, for example, one or moreobjects (e.g., walls, ceilings, furniture, stairs, cars, trees, trackingmarkers, or any other types of objects). The particular number ofsensors 348 may be fewer or more than the number of sensors depicted.The HMD device 344 may further include one or more additional componentsthat are not attached to the front-facing structure (e.g., are internalto the HMD device), such as an IMU (inertial measurement unit) 347electronic device that measures and reports the HMD device's 344specific force, angular rate, and/or the magnetic field surrounding theHMD device (e.g., using a combination of accelerometers and gyroscopes,and optionally, magnetometers). The HMD device may further includeadditional components that are not shown, including one or more displaypanels and optical lens systems that are oriented toward eyes (notshown) of the user and that optionally have one or more attachedinternal motors to change the alignment or other positioning of one ormore of the optical lens systems and/or display panels within the HMDdevice, as discussed in greater detail below with respect to FIG. 4.

The illustrated example of the HMD device 344 is supported on the headof user 342 based at least in part on one or more straps 345 that areattached to the housing of the HMD device 344 and that extend wholly orpartially around the user's head. While not illustrated here, the HMDdevice 344 may further have one or more external motors, such asattached to one or more of the straps 345, and automated correctiveactions may include using such motors to adjust such straps in order tomodify the alignment or other positioning of the HMD device on the headof the user. It will be appreciated that HMD devices may include othersupport structures that are not illustrated here (e.g., a nose piece,chin strap, etc.), whether in addition to or instead of the illustratedstraps, and that some embodiments may include motors attached one ormore such other support structures to similarly adjust their shapeand/or locations to modify the alignment or other positioning of the HMDdevice on the head of the user. Other display devices that are notaffixed to the head of a user may similarly be attached to or part ofone or structures that affect the positioning of the display device, andmay include motors or other mechanical actuators in at least someembodiments to similarly modify their shape and/or locations to modifythe alignment or other positioning of the display device relative to oneor more pupils of one or more users of the display device.

FIG. 4 illustrates a simplified top plan view 400 of an HMD device 405that includes a pair of near-to-eye display systems 402 and 404. The HMDdevice 405 may, for example, be the same or similar HMD devicesillustrated in FIGS. 1-3 or a different HMD device, and the HMD devicesdiscussed herein may further be used in the examples discussed furtherbelow. The near-to-eye display systems 402 and 404 of FIG. 4 includedisplay panels 406 and 408, respectively (e.g., OLED micro-displays),and respective optical lens systems 410 and 412 that each have one ormore optical lenses. The display systems 402 and 404 may be mounted toor otherwise positioned within a housing (or frame) 414, which includesa front-facing portion 416 (e.g., the same or similar to thefront-facing surface 343 of FIG. 3), a left temple 418, right temple 420and interior surface 421 that touches or is proximate to a face of awearer user 424 when the HMD device is worn by the user. The two displaysystems 402 and 404 may be secured to the housing 414 in an eye glassesarrangement which can be worn on the head 422 of a wearer user 424, withthe left temple 418 and right temple 420 resting over the user's ears426 and 428, respectively, while a nose assembly 492 may rest over theuser's nose 430. In the example of FIG. 4, the HMD device 405 may besupported on the head of the user in part or in whole by the nosedisplay and/or the right and left over-ear temples, although straps (notshown) or other structures may be used in some embodiments to secure theHMD device to the head of the user, such as the embodiments shown inFIGS. 2 and 3. The housing 414 may be shaped and sized to position eachof the two optical lens systems 410 and 412 in front of one of theuser's eyes 432 and 434, respectively, such that a target location ofeach pupil 494 is centered vertically and horizontally in front of therespective optical lens systems and/or display panels. Although thehousing 414 is shown in a simplified manner similar to eyeglasses forexplanatory purposes, it should be appreciated that in practice moresophisticated structures (e.g., goggles, integrated headband, helmet,straps, etc.) may be used to support and position the display systems402 and 404 on the head 422 of user 424.

The HMD device 405 of FIG. 4, and the other HMD devices discussedherein, is capable of presenting a virtual reality display to the user,such as via corresponding video presented at a display rate such as 30or 60 or 90 frames (or images) per second, while other embodiments of asimilar system may present an augmented reality display to the user.Each of the displays 406 and 408 of FIG. 4 may generate light which istransmitted through and focused by the respective optical lens systems410 and 412 onto the eyes 432 and 434, respectively, of the user 424.The pupil 494 aperture of each eye, through which light passes into theeye, will typically have a pupil size ranging from 2 mm (millimeters) indiameter in very bright conditions to as much as 8 mm in darkconditions, while the larger iris in which the pupil is contained mayhave a size of approximately 12 mm—the pupil (and enclosing iris) mayfurther typically move within the visible portion of the eye under openeyelids by several millimeters in the horizontal and/or verticaldirections, which will also move the pupil to different depths from theoptical lens or other physical elements of the display for differenthorizontal and vertical positions as the eyeball swivels around itscenter (resulting in a three dimensional volume in which the pupil canmove). The light entering the user's pupils is seen by the user 424 asimages and/or video. In some implementations, the distance between eachof the optical lens systems 410 and 412 and the user's eyes 432 and 434may be relatively short (e.g., less than 30 mm, less than 20 mm), whichadvantageously causes the HMD device to appear lighter to the user sincethe weight of the optical lens systems and the display systems arerelatively close to the user's face, and also may provide the user witha greater field of view. While not illustrated here, some embodiments ofsuch an HMD device may include various additional internal and/orexternal sensors.

In the illustrated embodiment, the HMD device 405 of FIG. 4 furtherincludes hardware sensors and additional components, such as to includeone or more accelerometers and/or gyroscopes 490 (e.g., as part of oneor more IMU units). As discussed in greater detail elsewhere herein,values from the accelerometer(s) and/or gyroscopes may be used tolocally determine an orientation of the HMD device. In addition, the HMDdevice 405 may include one or more front-facing cameras, such ascamera(s) 485 on the exterior of the front portion 416, and whoseinformation may be used as part of operations of the HMD device, such asfor providing AR functionality or positioning functionality.Furthermore, the HMD device 405 may further include other components 475(e.g., electronic circuits to control display of images on the displaypanels 406 and 408, internal storage, one or more batteries, positiontracking devices to interact with external base stations, etc.), asdiscussed in greater detail elsewhere herein. Other embodiments may notinclude one or more of the components 475, 485 and/or 490. While notillustrated here, some embodiments of such an HMD device may includevarious additional internal and/or external sensors, such as to trackvarious other types of movements and position of the user's body, eyes,controllers, etc.

In the illustrated embodiment, the HMD device 405 of FIG. 4 furtherincludes hardware sensors and additional components that may be used bydisclosed embodiments as part of the described techniques fordetermining user pupil or gaze direction, which may be provided to oneor more components associated with the HMD device for use thereby, asdiscussed elsewhere herein. The hardware sensors in this example includeone or more eye tracking assemblies 472 of an eye tracking subsystemthat are mounted on or near the display panels 406 and 408 and/orlocated on the interior surface 421 near the optical lens systems 410and 412 for use in acquiring information regarding the actual locationsof the user's pupils 494, such as separately for each pupil in thisexample.

Each of the eye tracking assemblies 472 may include one or more lightsources (e.g., IR LEDs) and one or more light detectors (e.g., siliconphotodiodes). Further, although only four total eye tracking assemblies472 are shown in FIG. 4 for clarity, it should be appreciated that inpractice a different number of eye tracking assemblies may be provided.In some embodiments, a total of eight eye tracking assemblies 472 areprovided, four eye tracking assemblies for each eye of the user 424.Further, in at least some implementations, each eye tracking assemblyincludes a light source directed at one of the user's 424 eyes 432 and434, a light detector positioned to receive light reflected by therespective eye of the user, and a polarizer positioned and configured toprevent light that is reflected via specular reflection from beingimparted on the light detector.

As discussed in greater detail elsewhere herein, information from theeye tracking assemblies 472 may be used to determine and track theuser's gaze direction during use of the HMD device 405. Furthermore, inat least some embodiments, the HMD device 405 may include one or moreinternal motors 438 (or other movement mechanisms) that may be used tomove 439 the alignment and/or other positioning (e.g., in the vertical,horizontal left-and-right and/or horizontal front-and-back directions)of one or more of the optical lens systems 410 and 412 and/or displaypanels 406 and 408 within the housing of the HMD device 405, such as topersonalize or otherwise adjust the target pupil location of one or bothof the near-to-eye display systems 402 and 404 to correspond to theactual locations of one or both of the pupils 494. Such motors 438 maybe controlled by, for example, user manipulation of one or more controls437 on the housing 414 and/or via user manipulation of one or moreassociated separate I/O controllers (not shown). In other embodimentsthe HMD device 405 may control the alignment and/or other positioning ofthe optical lens systems 410 and 412 and/or display panels 406 and 408without such motors 438, such as by use of adjustable positioningmechanisms (e.g., screws, sliders, ratchets, etc.) that are manuallychanged by the user via use of the controls 437. In addition, while themotors 438 are illustrated in FIG. 4 for only one of the near-to-eyedisplay systems, each near-to-eye display system may have its own one ormore motors in some embodiments, and in some embodiments one or moremotors may be used to control (e.g., independently) each of multiplenear-to-eye display systems.

While the described techniques may be used in some embodiments with adisplay system similar to that illustrated, in other embodiments othertypes of display systems may be used, including with a single opticallens and display device, or with multiple such optical lenses anddisplay devices. Non-exclusive examples of other such devices includecameras, telescopes, microscopes, binoculars, spotting scopes, surveyingscopes, etc. In addition, the described techniques may be used with awide variety of display panels or other display devices that emit lightto form images, which one or more users view through one or more opticallens. In other embodiments, the user may view one or more images throughone or more optical lens that are produced in manners other than via adisplay panel, such as on a surface that reflects light from anotherlight source in part or in whole.

FIG. 5 illustrates and example of using a plurality of eye trackingassemblies that each include light sources and light detectors todetermine a user's gaze location, in particular manners in particularembodiments in accordance with the described techniques. In particular,FIG. 5 includes information 500 to illustrate operation of an exampledisplay panel 510 and associated optical lens 508 in providing imageinformation to an eye 504 of a user, such as to focus the information ona pupil 506 of the eye. In the illustrated embodiment, four eye trackingassemblies 511a-511d, collectively 511, of an eye tracking subsystem aremounted proximate the edges of the optical lens 508 and are eachgenerally directed toward the pupil 506 of the eye 504 for emittinglight toward the eye 504 and capturing light reflected from the pupil506 or some or all the surrounding iris 502.

In the illustrated example, each of the eye tracking assemblies 511includes a light source 512, a light detector 514, and a polarizer 516positioned and configured to provide scattered light (diffusereflection) to the light detector while substantially prohibiting lightreflected by specular reflection from reaching the detector 514. In thisexample, the eye tracking assemblies 511 are placed at locations thatinclude near the top of the optical lens 508 along a central verticalaxis, near the bottom of the optical lens along the central verticalaxis, near the left of the optical lens along a central horizontal axis,and near the right of the display panel along the central horizontalaxis. In other embodiments, eye tracking assemblies 511 may bepositioned at other locations, and fewer or more eye tracking assembliesmay be used.

The polarizer 516 of each eye assembly 511 may include one or morepolarizers positioned in front of the light source 512 and/or the lightdetector 514 to reduce or eliminate specularly reflected light fromreaching the light detector 514. In one example, the polarizer 516 of aparticular eye tracking assembly 511 may include two cross-linearpolarizers, a first linear polarizer positioned in front of the lightsource 512 and a second linear polarizer oriented at 90 degrees withrespect to the first polarizer positioned in front of the light detector514 to block specular reflected light. In other implementations, one ormore circular or linear polarizers may be used to prevent lightreflected via specular reflection from reaching the light detectors 514,such that the light detectors substantially receive light reflected viadiffuse reflection.

It will also be appreciated that the light sources and light detectorsare shown for example purposes only, and that other embodiments mayinclude more or fewer light sources or detectors, and that the lightsources or detectors may be located in other locations. In addition,while not illustrated here, further hardware components may be used insome embodiments to assist in the acquisition of data from one or moreof the light detectors. For example, the HMD device or other displaydevice may include various light sources (e.g., infrared, visible light,etc.) at different positions to shine light on the iris and pupil to bereflected back to one or more light detectors, such as an light sourcemounted on or near the display panel 510, or instead elsewhere (e.g.,between the optical lens 508 and the eye 504, such as on an interiorsurface, not shown, of an HMD device that includes the display panel 510and optical lens 508). In some such embodiments, the light from such anillumination source may further be bounced off the display panel beforepassing through the optical lens 508 to illuminate the iris and pupil.

FIG. 6 is a schematic diagram of an environment 600 in which machinelearning techniques may be used to implement an eye tracking subsystemof an HMD device, such as the eye tracking subsystem discussed herein,according to one non-limiting illustrated implementation. Theenvironment 600 includes a model training portion 601 and an inferenceportion 603. In the training portion 601, training data 602 is fed intoa machine learning algorithm 604 to generate a trained machine learningmodel 606. The training data may include, for example, labeled data fromthe light detectors that specify gaze location. As a non-limitingexample, in an embodiment that includes four light detectors directed ata user's eye, each training sample may include the output from each ofthe four light detectors and a known or inferred gaze direction. In atleast some implementations, the gaze direction of a user may be known orinferred by directing a user to gaze at a particular user interfaceelement (e.g., a word, a dot, an “X”, another shape or object, etc.) ona display of the HMD device, which user interface element may be staticor may be moving on the display. The training data may also includesamples wherein one or more of the light sources or light detectors areoccluded, such as due to a user blinking, eye lashes, glasses, a hat, orother obstruction. For such training samples, the label may be “unknown”or “occluded” rather than a specified gaze direction.

The training data 602 may be obtained from a plurality of users and/orfrom a single user of an HMD system. The training data 602 may beobtained in a controlled environment and/or during actual use by user's(“field training”). Further, in at least some implementations, the model606 may be updated or calibrated from time-to-time (e.g., periodically,continuously, after certain events) to provide accurate gaze directionpredictions.

In the inference portion 603, run-time data 608 is provided as input tothe trained machine learning model 606, which generates gaze directionpredictions 610. Continuing with the above example, the output data ofthe light detectors may be provided as input to the trained machinelearning model 606, which may process the data to predict a gazelocation. The gaze direction predictions 610 may then be provided to oneor more components associated with and HMD device, such as, for example,one or more VR or AR applications executing on an HMD device, one ormore display or rendering modules, one or more mechanical controls, oneor more position tracking subsystems, etc.

The machine learning techniques employed to implement the featuresdiscussed herein may include any type of suitable structures ortechniques. As non-limiting examples, the machine learning model 606 mayinclude one or more of decision trees, statistical hierarchical models,support vector machines, artificial neural networks (ANNs) such asconvolutional neural networks (CNNs) or recurrent neural networks (RNNs)(e.g., long short-term memory (LSTM) networks), mixture density networks(MDNs), hidden Markov models, or others can be used. In at least someimplementations, such as implementations that utilize an RNN, themachine learning model 606 may utilize past input (memory, feedback)information to predict gaze direction. Such implementations mayadvantageously utilize sequential data to determine motion informationor previous gaze direction predictions, which may provide more accuratereal-time gaze direction predictions.

FIG. 7 is a flow diagram of an example embodiment of a method 700 ofoperating an eye tracking subsystem of an HMD device. The method 700 maybe performed by, for example, the eye tracking subsystem 135 of FIG. 1or other systems discussed elsewhere herein. While the illustratedembodiment of the 700 discusses performing operations to determine gazedirection for a single eye, it will be appreciated that the operationsof the method 700 may be applied to both eyes of a user simultaneouslyto track gaze direction in substantially real-time. It will also beappreciated that the illustrated embodiment of the method 700 may beimplemented in software and/or hardware as appropriate, and may beperformed by, for example, a computing system associated with an HMDdevice, for example.

As discussed above, the HMD device may include a support structurewearable on the head of a user, an eye tracking subsystem coupled to thesupport structure, at least one processor, ant memory storing a set ofinstructions or data. The eye tracking subsystem may include a pluralityof eye tracking assemblies that each include a light source, a lightdetector, and a polarizer.

The polarizer may be positioned proximate to at least one of the lightsource and the light detector and may be configured to prevent lightreflected via specular reflection from being received by the lightdetector. In at least some implementations, the polarizer may includetwo cross linear polarizers. In at least some implementations, for eachof the plurality of eye tracking assemblies, the polarizer includes afirst polarizer positioned in a light emitting path of the light sourceand a second polarizer positioned in a light detecting path of the lightdetector. More generally, the polarizer may include at least one of acircular polarizer or a linear polarizer.

Each of the light sources may be directed at a target location of auser's pupil, which may allow for lower powered light sources to be usedsince the energy is focused on the target location. Additionally,various optics (e.g., lenses) or types of light sources (e.g., IRlasers) may also be used to focus light on a target area. In at leastsome implementations, the light sources may include light emittingdiodes that emit light having a wavelength that is between 780 nm and1000 nm, for example. The light sources may be positioned away from anoptical axis of an eye of the user to provide dark field illumination ofthe pupil. In at least some implementations, the light detectors mayinclude silicon photodiodes that provide an output signal dependent onincident light power.

The illustrated embodiment of the method 700 begins at 702, where atleast one processor of a HMD device may selectively cause the lightsources of the plurality of eye tracking assemblies to emit light. Theat least one processor may cause the light sources to emit lightsimultaneously, sequentially, in another pattern, or any combinationthereof. At 704, the at least one processor may receive light detectioninformation captured by the light detectors of the plurality of eyetracking assemblies. For example, the at least one processor may storeoutput data received from a plurality of light detectors at one or moretime periods.

At 706, the at least one processor may provide the received lightdetection information as input to a trained machine learning model, suchas the model 606 discussed above in relation to FIG. 6. As discussedabove, the machine learning model may include an RNN, an MDN, or anyother type of machine learning model that is suitable to provideaccurate gaze direction predictions based on light data input receivedfrom a plurality of light detectors. As discussed elsewhere herein, inother implementations a prediction model or function other than amachine learning model may be used, such as a 1D or 2D polynomial, oneor more lookup tables, etc.

At 708, the at least one processor may receive, from the machinelearning model in response to providing the light detection information,a determined gaze direction for an eye of the user. The determined gazedirection may be provide in any suitable format.

To calibrate or update the machine learning model, the at least oneprocessor may cause at least one display of the HMD to present a userinterface element, selectively cause the light sources of the pluralityof eye tracking assemblies to emit light, and receive light detectioninformation captured by the light detectors of the plurality of eyetracking assemblies. The light detection information, along withcorresponding known or inferred gaze direction information, may be usedto update the machine learning model. The model may be updated a varioustimes as required to provide accurate gaze direction predictions. Theuser interface element may include a static user interface element or amoving user interface element.

At 710, the at least one processor provide the determined gaze directionto a component associated with the HMD system for use thereby. Forexample, the determined gaze direction may be provided to an imagerendering subsystem of the HMD system to provide foveated renderingbased on the determined gaze direction, as discuss above. As anotherexample, the eye tracking subsystem may determine that the user's eyesare saccading, and may dynamically modify image rendering to takeadvantage of saccadic masking or saccadic suppression. For example, oneor more characteristics of image rendering may be modified duringsaccading, such as the resolution of all or portions of images, thespatial frequencies of images, the frame rate, or any othercharacteristic that may allow for reduced bandwidth, lower computationrequirements, or other technical benefits.

As another example, in at least some implementations the HMD device mayinclude an interpupillary distance (IPD) adjustment component that isoperative to automatically adjust one or more component of the HMD toaccount for variable IPD. In this example, the IPD adjustment componentmay receive gaze direction, and may align at least one component of theHMD system for the user based at least in part on the determined gazedirection. For example, when a user is looking at nearby objects, theIPD may be relatively shorter and the IPD adjustment may align one ormore components of the HMD device accordingly. As yet anothernon-limiting example, the HMD device may automatically adjust a focus ofa lens based on the determined gaze direction.

Although the examples above utilize machine learning techniques todetermine gaze direction from light detection information, it should beappreciated that the features of the present disclosure are not limitedto using machine learning techniques.

Generally, any type of prediction model or function may be used. Forexample, in at least some implementations, rather than going directlyfrom light detection information to gaze direction, the system may dothe opposite—predict light detection information given an input gazedirection. Such methods may find a prediction function or model thatperforms this prediction, mapping from a gaze direction to predictedlight readings for that direction. This function may be user-specific,so the system may implement a calibration process to find or customizethe function. Once the prediction function has been determined orgenerated, the prediction function may then be inverted (e.g., with anumerical solver) to produce real-time predictions during use.Specifically, given a sample of light detection information from realsensors, the solver is operative to find the gaze direction thatminimizes the error between this real reading and predicted readingsfrom the generated prediction function. In such implementations, theoutput is the solved gaze direction plus a residual error, which couldbe used to judge the quality of the solution. In at least someimplementations, some additional correction may be applied to handlevarious issues, such as the HMD system sliding around on the user's faceduring operation. The prediction function may be any type of function.As an example, a set of 2D polynomials may be used to map gaze angles tophotodiode readings given a dataset of points captured from a user. Inat least some other implementations, lookup tables or other approaches,including fitting a ML system to output predictions as discussed above,may also be used.

It will be appreciated that in some embodiments the functionalityprovided by the routines discussed above may be provided in alternativeways, such as being split among more routines or consolidated into fewerroutines. Similarly, in some embodiments illustrated routines mayprovide more or less functionality than is described, such as when otherillustrated routines instead lack or include such functionalityrespectively, or when the amount of functionality that is provided isaltered. In addition, while various operations may be illustrated asbeing performed in a particular manner (e.g., in serial or in parallel)and/or in a particular order, those skilled in the art will appreciatethat in other embodiments the operations may be performed in otherorders and in other manners. It will similarly be appreciated that thedata structures discussed above may be structured in different manners,including for databases or user interface screens/pages or other typesof data structures, such as by having a single data structure split intomultiple data structures or by having multiple data structuresconsolidated into a single data structure. Similarly, in someembodiments illustrated data structures may store more or lessinformation than is described, such as when other illustrated datastructures instead lack or include such information respectively, orwhen the amount or types of information that is stored is altered.

In addition, the sizes and relative positions of elements in thedrawings are not necessarily drawn to scale, including the shapes ofvarious elements and angles, with some elements enlarged and positionedto improve drawing legibility, and the particular shapes of at leastsome elements being selected for ease of recognition without conveyinginformation regarding the actual shape or scale of those elements. Inaddition, some elements may be omitted for clarity and emphasis.Furthermore, repeated reference numbers in different drawings maysignify the same or similar elements.

From the foregoing it will be appreciated that, although specificembodiments have been described herein for purposes of illustration,various modifications may be made without deviating from the spirit andscope of the invention. In addition, while certain aspects of theinvention are presented at times in certain claim forms, or may not beembodied in any claims at some times, the inventors contemplate thevarious aspects of the invention in any available claim form. Forexample, while only some aspects of the invention may be recited at aparticular time as being embodied in a computer-readable medium, otheraspects may likewise be so embodied.

1. A head-mounted display (HMD) system, comprising: a support structurewearable on the head of a user; an eye tracking subsystem coupled to thesupport structure, the eye tracking subsystem comprising a plurality ofeye tracking assemblies that each include: a light source operative toemit light; a light detector operative to detect light; and a polarizerpositioned proximate to at least one of the light source and the lightdetector, the polarizer configured to prevent light reflected viaspecular reflection from being received by the light detector whileallowing light reflected via diffuse reflection to be received by thelight detector, wherein light reflected via specular reflectioncomprises light that is reflected from a surface at an angle equal tothe angle of incidence, and light reflected via diffuse reflectioncomprises light that is scattered from a surface at many angles; atleast one processor; and memory storing a set of instructions or datathat, as a result of execution, cause the HMD system to: selectivelycause the light sources of the plurality of eye tracking assemblies toemit light; receive light detection information captured by the lightdetectors of the plurality of eye tracking assemblies; provide thereceived light detection information as input to a prediction model;receive, from the prediction model in response to providing the lightdetection information, a determined gaze direction for an eye of theuser; and provide the determined gaze direction to a componentassociated with the HMD system for use thereby.
 2. The HMD system ofclaim 1 wherein the light detection information comprises a signatureradiation pattern of each of the light sources after the light from thelight source has reflected, scattered, or been absorbed of off the faceor eye of the user.
 3. The HMD system of claim 1 wherein each of thelight sources is directed toward an expected location of a pupil of theuser.
 4. The HMD system of claim 1 wherein the light sources compriselight emitting diodes that emit light having a wavelength that isbetween 780 nm and 1000 nm.
 5. The HMD system of claim 1 wherein thelight detectors comprise photodiodes.
 6. The HMD system of claim 1wherein the eye tracking subsystem comprises four eye trackingassemblies positioned to determine the gaze direction of a left eye ofthe user and four eye tracking assemblies positioned to determine thegaze direction of a right eye of the user.
 7. The HMD system of claim 1wherein the polarizer comprises two cross linear polarizers.
 8. The HMDsystem of claim 1 wherein, for each of the plurality of eye trackingassemblies, the polarizer comprises a first polarizer positioned in alight emitting path of the light source and a second polarizerpositioned in a light detecting path of the light detector.
 9. The HMDsystem of claim 1 wherein the polarizer comprises at least one of acircular polarizer or a linear polarizer.
 10. The HMD system of claim 1wherein, for each of the plurality of eye tracking assemblies, the lightsource is positioned away from an optical axis of an eye of the user toprovide dark field illumination of the pupil.
 11. The HMD system ofclaim 1 wherein the prediction model comprises a machine learning model.12. The HMD system of claim 11 wherein the machine learning modelcomprises a mixture density network (MDN) model.
 13. The HMD system ofclaim 11 wherein the machine learning model comprises a recurrent neuralnetwork (RNN) model.
 14. The HMD system of claim 11 wherein the machinelearning model utilizes past input information or eye motion informationto determine the gaze direction.
 15. The HMD system of claim 11 whereinthe machine learning model is a model trained during field operation ofa plurality of HMD systems.
 16. The HMD system of claim 1 wherein theprediction model comprises a polynomial or a lookup table.
 17. The HMDsystem of claim 1 wherein the HMD system comprises at least one display,and the at least one processor: causes the at least one display topresent a user interface element; selectively causes the light sourcesof the plurality of eye tracking assemblies to emit light; receiveslight detection information captured by the light detectors of theplurality of eye tracking assemblies; and updates the machine learningmodel based at least in part on the received light detection informationand known or inferred gaze direction information associated with thereceived light detection information.
 18. The HMD system of claim 17wherein the user interface element includes a static user interfaceelement or a moving user interface element.
 19. The HMD system of claim1 wherein the HMD system comprises at least one display, and the atleast one processor: dynamically modifies a rendering output of the atleast one display based at least in part on the determined gazedirection.
 20. The HMD system of claim 1 wherein the HMD systemcomprises an interpupillary distance (IPD) adjustment component, and theat least one processor: causes the IPD adjustment component to align atleast one component of the HMD system for the user based at least inpart on the determined gaze direction.
 21. A method of operating ahead-mounted display (HMD) system, the HMD system comprising an eyetracking subsystem coupled to a support structure that comprises aplurality of eye tracking assemblies that each include a light source, alight detector, and a polarizer, the method comprising: selectivelycausing the light sources of the plurality of eye tracking assemblies toemit light; preventing, by each of the respective polarizers of theplurality of eye tracking assemblies, light reflected via specularreflection from being received by the plurality of light detectors whileallowing light reflected via diffuse reflection to be received by theplurality of light detectors, wherein light reflected via specularreflection comprises light that is reflected from a surface at an angleequal to the angle of incidence, and light reflected via diffusereflection comprises light that is scattered from a surface at manyangles; receiving light detection information captured by the pluralityof light detectors, the light detection information comprising the lightreflected by diffuse reflection; providing the received light detectioninformation as input to a trained machine learning model; receiving,from the trained machine learning model in response to providing thelight detection information, a determined gaze direction for an eye of auser; and providing the determined gaze direction to a componentassociated with the HMD system for use thereby.
 22. The method of claim21 wherein the light detection information comprises a signatureradiation pattern of each of the light sources after the light from thelight source has reflected, scattered, or been absorbed of off the faceor eye of the user.
 23. The method of claim 21 wherein providing thereceived light detection information as input to the trained machinelearning model comprises providing the received light detectioninformation as input to a mixture density network (MDN) model.
 24. Themethod of claim 21 wherein providing the received light detectioninformation as input to the trained machine learning model comprisesproviding the received light detection information as input to arecurrent neural network (RNN) model.
 25. The method of claim 21 whereinproviding the received light detection information as input to thetrained machine learning model comprises providing the received lightdetection information to a machine learning model that utilizes pastinput information or eye motion information to determine the gazedirection.
 26. The method of claim 21, further comprising training thetrained machine learning model during field operation of a plurality ofHMD systems.
 27. The method of claim 21 wherein the HMD system comprisesat least one display, and the method comprises: causing the at least onedisplay to present a user interface element; selectively causing thelight sources of the plurality of eye tracking assemblies to emit light;receiving light detection information captured by the plurality of lightdetectors; and updating the trained machine learning model based atleast in part on the received light detection information and known orinferred gaze direction information associated with the received lightdetection information.
 28. The method of claim 27 wherein causing the atleast one display to present a user interface element comprises causingthe at least one display to present a static user interface element or amoving user interface element.
 29. The method of claim 21 wherein theHMD system comprises at least one display, and the method comprises:dynamically modifying an output of the at least one display based atleast in part on the determined gaze direction.
 30. The method of claim21, further comprising: mechanically aligning at least one component ofthe HMD system for the user based at least in part on the determinedgaze direction.
 31. A head-mounted display (HMD) system, comprising: asupport structure wearable on the head of a user; an eye trackingsubsystem coupled to the support structure, the eye tracking subsystemcomprising a plurality of eye tracking assemblies that each include: alight emitting diode; a photodiode; and a polarizer positioned proximateto at least one of the light emitting diode and the photodiode, thepolarizer configured to prevent light reflected via specular reflectionfrom being received by the photodiode while allowing light reflected viadiffuse reflection to be received by the photodiode, wherein lightreflected via specular reflection comprises light that is reflected froma surface at an angle equal to the angle of incidence, and lightreflected via diffuse reflection comprises light that is scattered froma surface at many angles; at least one processor; and memory storing aset of instructions or data that, as a result of execution, cause theHMD system to: selectively cause the light emitting diodes of theplurality of eye tracking assemblies to emit light; receive lightdetection information captured by the photodiodes of the plurality ofeye tracking assemblies; provide the received light detectioninformation as input to a prediction model; receive, from the predictionmodel in response to providing the light detection information, adetermined gaze direction for an eye of the user; and dynamically modifyoperation of a component associated with the HMD system based at leastin part on the determined gaze direction.